Hide and seek: worms digging at the Internet backbones and edges

The problem of malicious shellcode detection in high-speed network channels is a significant part of the more general problem of botnet propagation detection and filtering. Many of the modern botnets use remotely exploitable vulnerabilities in popular networking software for automatic propagation. We formulate the problem of shellcode detection in network flow in terms of formal theory of heuristics combination, where a set of detectors are used to recognize specific shellcode features and each of the detectors has its own characteristics of shellcode space coverage, false negative and false positive rates and computational complexity. Since the set of detectors and their quality is the key to the problem’s solution, we will provide a survey of existing shellcode detection methods, including static, dynamic, abstract execution and hybrid, giving an estimation to the quality of the characteristics for each of the methods. Keywords-shellcode; malware; polymorphism; metamorphism; botnet detection;